Deciding on Metrics

Re: Unit of Diversion

Three main categories of diversion were presented in the course: event-based
diversion, cookie-based diversion, and account-based diversion.

An event-based diversion (like a pageview) can provide many observations to
draw conclusions from, but doesn't quite hit the mark for this case. If the
condition changes on each pageview, then a visitor might get a different
experience on each homepage visit. Event-based diversion is much better when
the changes aren't as easily visible to users, to avoid disruption of
experience. In addition, pageview-based diversion would let us know how many
times the download page was accessed from each condition, but can't go any
further in tracking how many actual downloads were generated from each
condition.

Diverting based on account or user ID can be stable, but it's not the right
choice in this case. Since visitors only register after getting to the download
page, this is too late to introduce the new homepage to people who should be
assigned to the experimental condition.

So this leaves the consideration of cookie-based diversion, which feels like
the right choice. We can assign a cookie to each visitor upon their first page
hit, which allows them to be separated into the control and experimental groups.
Cookies also allow tracking of each visitor hitting each page, recording whether or
not they eventually hit the download page and then whether or not they actually register an
account and perform the download.

That's not to say that the cookie-based diversion is perfect. The usual
cookie-based diversion issues apply: we can get some inconsistency in counts
if users enter the site via incognito window, different browsers, or cookies
that expire or get deleted before they make a download. This kind of assignment
'dilution' could dampen the true effect of our experimental manipulation. As a
simplification, however, we'll assume that this kind of assignment dilution
will be small, and ignore its potential effects.

Re: Brainstorm Potential Metrics

In terms of metrics, we might want to keep track of the number of cookies that
are recorded in different parts of the website. In particular, the number of
cookies on the homepage, download page, and account registration page (in order
to actually make the download) could prove useful. We can track the number of
licenses purchased through the user accounts, each of which can be linked back
to a particular condition. Though it hasn't been specified, it's also possible
that the software includes usage statistics that we could track.

The above metrics are all based on absolute counts. We could instead
perform our analysis on ratios of those counts. For example, we could be
interested in the proportion of downloads out of all homepage visits.
License purchases could be stated as a ratio against the number of registered
users (downloads) or the original number of cookies.

Below, you will decide for each of the proposed metrics whether or not you
would want to use them as an invariant metric or an evaluation metric. To recap,
an invariant metric is an objective measure that you should expect will not
vary between conditions and that indicate equivalence between groups.
Evaluation metrics, on the other hand, represent measures where you expect there will be
differences between groups, and whose differences should say something
meaningful about your experimental manipulation.

Selecting Invariant and Evaluation Metrics

QUIZ QUESTION::

For each metric below, select whether you would use it as an invariant metric, evaluation metric, or neither.

ANSWER CHOICES:

Proposed Metric	Metric Type
Number of cookies @ homepage
Number of cookies @ download page
Number of user ids / downloads
Number of license purchases
Mean software usage time during trial
Ratio: # downloads / # cookies
Ratio: # licenses / # cookies
Ratio: # licenses / # user IDs

SOLUTION:

Proposed Metric	Metric Type
Number of cookies @ download page
Number of user ids / downloads
Number of license purchases
Mean software usage time during trial
Ratio: # licenses / # user IDs
Ratio: # downloads / # cookies
Ratio: # licenses / # cookies
Number of cookies @ download page
Number of user ids / downloads
Number of license purchases
Mean software usage time during trial
Ratio: # licenses / # user IDs
Number of cookies @ homepage
Ratio: # downloads / # cookies
Ratio: # licenses / # cookies
Number of cookies @ download page
Number of user ids / downloads
Number of license purchases
Mean software usage time during trial
Ratio: # licenses / # user IDs
Number of cookies @ download page
Number of user ids / downloads
Number of license purchases
Mean software usage time during trial
Ratio: # licenses / # user IDs
Ratio: # downloads / # cookies
Ratio: # licenses / # cookies
Number of cookies @ download page
Number of user ids / downloads
Number of license purchases
Mean software usage time during trial
Ratio: # licenses / # user IDs
Number of cookies @ download page
Number of user ids / downloads
Number of license purchases
Mean software usage time during trial
Ratio: # licenses / # user IDs
Number of cookies @ homepage
Number of cookies @ homepage

Next Concept